A0-A087  684  hONEYFELL  SYSTEMS  AND  RESEARCH  CENTER  MINNEAPOLIS  NN  F/Q  17/7 

ADVANCED  PATTERN-MATChINO  TECHNIQUES  FOR  AUTONOMOUS  ACQUISITION— ETC (U) 
mAR  80  P  M  NARENDRA*  J  J  GRABAU  DAAK70-79-C-01H 

UNCLASSIFIED  80SRC25  NL 


ADVANCED  PATTERN-MATCHING  TECHNIQUES 

FOR 

AUTONOMOUS  ACQUISITION 

Siuri  lurttrly  Pnpriss  Kipirt 


NOTICE 


When  Government  drawings,  specifications,  or  other  data  are  used  for  any 
>  purpose  other  than  in  connection  with  a  definitely  related  Government  pro¬ 
curement  operation,  the  United  States  Government  thereby  incurs  no  respon¬ 
sibility  nor  any  obligation  whatsoever;  and  the  fact  that  the  government  may 
have  formulated,  furnished,  or  in  any  way  supplied  the  said  drawings,  spec¬ 
ifications,  or  other  data,  is  not  to  be  regarded  by  implication  or  otherwise 
as  in  any  manner  licensing  the  holder  or  any  other  person  or  corporation, 
or  conveying  any  rights  or  permission  to  manufacture,  use,  or  sell  any 
patented  invention  that  may  in  any  way  be  related  thereto. 


Copies  of  this  report  should  not  be  returned  unless  return  is  required  by 
security  considerations,  contractual  obligations,  or  notice  on  a  specific 
document. 


i 


maMMMfiiiiiM 


m 


UNCLASSIFIED 


SECURITY  CLASSIFICATION  OF  this  PACE  TI*A»r.  Dmlm  Fntrtril) 


REPORT  DOCUMENTATION  PAGE 


RKAD  INSTRUCTION'S 
BEFORE  COMPI.F-TINO  FORM 


>  AEPOPT  NUMBER 


/ 


4  T*?l€  fmnd  GubtiTte) 


I?  GOVT  accession  NO 


5  RECIPIENT'S  CAT  ALOG  NUMBF  R 


ADVANCED  PATTERN-MATCHING 
TECHNIQUES  FOR  AUTONOMOUS 
ACQUISITION 


/  / 


23  Nov,y4§79  - - 

j.21  FeblMmT _ _ 

*■  P^RFORMINyOWSTREPORT  NUMBER 


8oSRC25 


7  AUTHORC*) 


a  CONTRACT  or  grant  nuMBER'm 


j  P.  M./Narendra 
I  J.  J./Grabau 


/ 


/ 

✓  I 


DA  AK70-  79-C  -01 1 4?“' 


a  performing  organization  name  ano  address 

Honeywell  Inc.,  Systems  and  Research  Center 
2600  Ridgway  Parkway 
Minneapolis,  Minnesota  55413 


10.  PROGRAM  ELEMENT.  PROJECT  TASK 
AREA  A  WORK  UNIT  NUMBERS 


n  controlling  orpicc  name  ano  address 


12  REPORT  DATE 


Night  Vision  and  Electro-Optics  Laboratory  •-( 
Fort  Belvoir,  Virginia  22060 

/ 


U MONITORING  AGENCY  NAME  a  AOORESSri71i7fer#nf  from  Controlling  Ofhcf) 


MarBfc?f380 

T_l-  _MiiMpPP  r>F  Ptr.rt 

i£L 


is.  security  class.  '«»<  fin*  r«»i»<rf 


Unclassified 


15.  DECLASSIFICATION  DOWNGRADIN' 
SCHEDULE 


i*  distribution  statement  mi  i*i,  Htporn 

Unlimited 


ThU  do*  ."•'SNVi  o-<’d 

for  public  i  '  i  •  "■  5 

distribution  is  : ■  i’.oci. 


17  DISTRIBUTION  STATEMENT  (ol  the  mbstrmci  entered  in  Htock  JO,  if  ditterenf  from  Report) 


*•  supplementary  notes 


Mr.  C.  Goehrig  is  NV&EOL  Contract  Monitor  on  this  program. 


19  KF  f  «OR0S  '  Lixifinuf  *tde  •/  end  identity  Av  block  nu ntbrt) 

Infrared  Target  recognition  Symbolic  processing 

FLIR  Pattern  recognition  Target  tracking 

Target  cueing  Image  processing  Scene  analysis 

Target  screening  Artificial  intelligence  Pattern  matching 


AHM  RA'  r  'Continue  r.n  re*-er%m  *tde  II  nece  %»»r\  i w»<  •  ’entity  A»  hloi  A  m  P.h«l 

This  is  the  second  interim  quarterly  progress  report  on  "Advanced 
Pattern- Matching  Concepts,"  NV&EOL  contract  No.  DAAK70-79-C-01 14. 
It  reports  the  results  of  work  performed  between  November  23,  1979  and 
February  23,  1980. 

The  key  objective  of  this  effort  is  the  development  of  pattern- matching 
^lgoritiun£^hich<iCan>Hnpart^utonomous^c2uisition<ca£abilityt^^_i^^^ 


DD  1  JAN  71  1473  EOITION  of  I  NOV  55  IS  OBSOLETE 


■UNCLASSIFIED  U  , _ 

SECURITY  Cl  ASSfEiC  ATiON  t>  This  PAGE  t'miH  *  - 


I 


L  ASSIGNATION  OF  Tmi$  PAGEr»»'hen  hniermd) 


20.  ABSTRACT  (continued) 

precision- guided  munitions  such  as  Copperhead  and  Hellfire.  Autonomous 
acquisition  through  pattern  matching  holds  the  promise  of  eliminating 
laser  designation  and  enhancing  fire  power  by  multiple  target  prioritization. 

The  pattern- matching  approach  being  developed  under  this  program  is 
based  on  a  symbolic  pattern-matching  framework,  which  is  suited  for  the 
autonomous  acquisition  scenario.  It  is  based  on  matching  a  symbolic 
representation  derived  from  the  two  images,  and  it  can  accommodate  the 
stringent  pattern- matching  criteria  established  by  the  scenario:  enormous 
differences  in  the  scene  perspective,  aspect  and  range  between  the  two 
sensors,  differences  in  sensor  characteristics  and  illumination,  and  scene 
changes  such  as  target  motion  and  obscuration  from  one  view  point  to  the 
other. 


.  .  xC'->  i 


'.Tv-.vii01' 


UNCLASSIFIED 


■>CCu*MT*  CLASSIFICATION  OF  This  P  A  C.F  'WTifft  tu !*•  i  nt  +  rrt 


CONTENTS 


Section 

1  INTRODUCTION 
Overview  of  Pattern  Matching 
Summary  of  Progress 
Report  Organization 

2  CANDIDATE  MATCH  SELECTION 

3  BRANCH- AND-BOUND  MATCHING  ALGORITHM 

4  CRITERION  FUNCTIONS 

The  Distinctiveness  Component 
The  Object  Similarity  Component 
The  Configuration  Component 

5  SYSTEM  SIMULATION  AND  RESULTS 
Example  1 

Example  2 


Example  3 


CONTENTS  (concluded) 


Section  Page 

6  ANALYSIS  OF  A  PRIORI  INFORMATION  76 

7  MINIMAL  SPANNING  TREES  80 

8  PATTERN-MATCHING  DATA  BASE  87 

9  PLANS  FOR  THE  NEXT  REPORTING  PERIOD  95 


iv 


r 

i 

1 

> 

! 

LIST  OF  ILLUSTRATIONS 

1 

1 

! 

• 

■« 

Figure 

•  Page  S 

1 

Pattern-Matching  Overview 

2  ! 

* ' 

2 

Block  Diagram  of  Pattern- Matching  Process 

1 

8 

3 

FLIR  Images  for  Example 

* 

10  : 

4 

Objects  Extracted  for  Images  of  Figure  3 

\ 

12  | 

(i 

5 

Candidate  Match  Selection  Example 

13  | 

6 

Illustrates  the  Sensor  Geometry  for  the 

Example  in  Figure  3 

21  i 

7 

Example  of  a  Branch- and- Bound  Search  Tree 

25 

8 

Branch-and-Bound  Algorithm  Flow  Chart 

28 

9 

A  Prioritized  Branch-and-Bound  Search  Tree 

37 

10 

An  Example  Showing  How  the  Function  f 

Detects  Bad  Matches  ms 

• 

43 

11 

Examples  of  Topologically  Consistent  and 
Topologically  Inconsistent  Matches 

46 

12 

Block  Diagram  of  Simulation  Software 

50  1 

13 

A  Complete  Search  Tree  for  Example  1 

54  | 

« 

14 

The  First  14  Matches  Evaluated  in  Example  1 

55 

15 

The  First  Four  Matches  Evaluated  in  Example  2 

71 

V 

1 

LIST  OF  ILLUSTRATIONS  (concluded) 


Figure  Page 

16  A  Minimal  Spanning  Tree  for  a  Graph  81 

17  A  Minimal  Spanning  Mat  Clusters  When  it  is  Broken  82 

18  Minimal  Spanning  Trees  in  Images  of  Figure  3  83 

19  Some  FLIR  Images  from  the  AP  Hill  Data  Base  88 


vi 


LIST  OF  TABLES 


Table  Page 

1  Example  of  Editing  Matches  in  Step  2 

of  the  Candidate  Match  Selection  Algorithm  11 

2  An  Example  of  Ranking  Acceptable  Matches  11 

3  Example  Candidate  Match  Selection  19 

4  Candidate  Match  Selection  24 

5  Branch-and-Bound  Algorithm  Parameters  for  Example  1  53 

6  Candidate  Match  Selection  for  Example  2  70 

7  Data  for  Example  3  75 

8  Current  Data  Base  Summary  94 


vii 


SECTION  1 


INTRODUCTION 


This  is  the  second  interim  quarterly  progress  report  on  "Advanced  Pattern- 
Matching  Concepts,  "  NV&EOL  contract  No.  DAAK70-79-C-0114.  It 
reports  the  results  of  work  performed  between  November  23,  1979  and 
February  23,  1980. 

The  key  objective  of  this  effort  is  the  development  of  pattern- matching 
algorithms  which  can  impart  autonomous  acquisition  capability  to  precision- 
guided  munitions  such  as  Copperhead  and  Hellfire.  Autonomous  acquisition 
through  pattern  matching  holds  the  promise  of  eliminating  laser  designation 
and  enhancing  fire  power  by  multiple- target  prioritization.  However,  this 
application  imposes  stringent  performance  requirements  on  the  pattern- 
matching  algorithm--it  must  be  robust  under  perspective  and  aspect 
change,  target  motion,  illumination  change,  sensor  differences,  image 
quality,  and  target  obscuration.  Conventional  pattern- matching  techniques 
are  incapable  of  meeting  the  performance  requirements  in  the  autonomous 
acquisition  scenario. 

The  pattern-matching  approach  being  developed  under  this  program  is  based 
on  a  symbolic  pattern- matching  framework  which  is  suited  for  the  autonomous 
acquisition  scenario.  It  is  based  on  matching  a  symbolic  representation 
derived  from  the  two  images,  rather  than  on  numerical  correlation.  It  can 


accommodate  the  stringent  pattern- matching  criteria  established  by  the 
scenario:  enormous  differences  in  the  scene  perspective,  aspect  and 
range  between  the  two  sensors,  differences  in  sensor  characteristics 
and  illumination,  and  scene  changes  such  as  target  motion  and  obscuration 
from  one  view  point  to  the  other. 

Figure  1  shows  a  broad  overview  of  the  symbolic  pattern- matching  approach. 
The  symbolic  pattern- matching  technique  is  based  on  matching  a  symbolic 
representation  of  the  two  images,  not  the  gray  levels  of  the  individual 
picture  elements  themselves,  as  in  conventional  correlation  approaches. 

A  symbolic  representation  of  an  image  consists  of  describing  objects  (or 
distinctive  elements)  in  the  image  and  their  positions. 


Figure  1.  Pattern-Matching  Overview 


The  symbolic  matching  technique  operates  on  the  symbolic  image  to  find 
an  optimal  match  which  simultaneously  ensures  1)  that  the  matched  objects 
are  similar  in  their  descriptors  and  2)  that  the  interobject  configuration 
in  the  two  matched  sets  of  objects  are  consistent.  This  yields  a  robust 
pattern- matching  algorithm  which  is  insensitive  to  a  number  of  variables, 
including  variation  in  object  descriptors. 

OVERVIEW  OF  PATTERN  MATCHING 

Figure  1  shows  the  basic  steps  in  symbolic  pattern  matching  between  the 
reference  and  sensed  images.  The  first  step  is  to  extract  object  features 
from  both  images.  Note  that  the  result  of  this  scene  analysis  combines 
both  blobs  (from  a  target  screener  segmenter)  and  edge- based  features 
such  as  long  lines  and  vertices  as  well.  The  next  stage  is  the  symbolic 
description  of  these  objects  (blobs  and  lines)  as  lists  of  object  descriptors 
and  their  positions  in  the  two  FOV.  Finally,  symbolic  pattern  matching 
is  performed  between  the  two  symbolic  images  to  establish  correspondence 
between  the  target  designated  in  the  reference  image  and  the  corresponding 
object  in  the  sensed  image. 

The  key  to  a  robust  symbolic  matching  is  that  in  an  optimal  match  both 
the  object  descriptions  and  the  inter- object  relationships  in  the  two  images 
should  be  consistent.  Criteria  for  evaluating  interobject  configuration 
matches  must  be  able  to  account  for  scene  perspective,  aspect,  roll, 
scale  differences.  The  development  of  an  efficient  search  algorithm  to 
examine  the  most  promising  object  and  configuration  matches  as  defined  by 
the  match  criteria  is  the  main  thrust  of  this  program. 
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SUMMARY  OF  PROGRESS 

The  following  is  a  summary  of  the  progress  made  in  the  second  quarter 
toward  the  above  program  objective. 

1.  Algorithms  for  candidate  match  selection  have  been  developed, 
implemented,  and  evaluated  with  FLIR  imagery.  Examples 
indicate  that  the  candidate  match  selection  process  is  able  to 
limit  the  number  of  candidates  to  a  few  without  discarding  any 
correct  object- to-object  matches.  The  procedure  is  illustrated 
with  an  example  taken  from  a  pair  of  FLIR  images. 

2.  Given  the  candidate  matches,  an  efficient  branch- and-bound 
enumeration  scheme  to  find  the  best  match  of  object  subsets 
in  the  two  fields  of  view  has  been  developed  and  implemented. 
Prioritized  and  unprioritized  versions  of  an  enumeration  scheme 
are  illustrated  with  examples.  The  algorithm  finds  the  best  match 
without  an  exhaustive  enumeration  of  all  possible  matches. 

3.  Several  criterion  functions  have  been  specified  for  use  in  the 
branch-and-bound  algorithm.  Criterion  functions  are  measures 
of  the  goodness  of  an  object  configuration  match  between  the  two 
images.  The  criteria  include  the  similarity  of  individual  object- 
to-object  matches,  geometric  transformation  consistency,  and 
topological  consistency.  It  is  shown  that  the  criteria  are  monotonic, 
guaranteeing  that  the  branch-and-bound-algorithm  yields  the 
optimal  solution  among  all  possible  solutions. 
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4.  All  components  of  the  matching  process— the  object  extraction, 
object  feature  extraction,  candidate  match  selection,  and  the  branch- 
and-bound  algorithm  (with  the  criterion  functions)— have  been 
installed  in  a  complete  system  simulation  at  the  Honeywell  Image 
Processing  facility.  Two  pattern- matching  examples  on  FLER 
imagery  obtained  with  the  system  simulation  are  included.  The 
examples  demonstrate  that  correct  pattern  matching  is  feasible 
between  two  fields  of  view  50  deg  apart,  even  when  only  a  few 
objects  are  common  to  both  fields  of  view. 

5.  A  systematic  analysis  of  how  a  priori  information  can  be  utilized  to 
speed  up  the  search  process  has  been  made.  A  computer  program 

to  predict  the  efficiency  of  the  algorithm  utilizing  a  priori  information 
has  been  coded  and  tested. 

6.  A  minimal  spanning  tree  approach  to  identifying  distinctive  clusters 
of  objects  in  the  two  fields  of  view  has  been  developed.  The 
technique  appears  to  have  the  potential  of  speeding  up  the  search 
process  for  the  optimum  match  by  finding  good  initial  object- to- 
object  matches,  using  similar  distinctive  object  clusters. 

7.  The  data  base  generation  task  continues;  36  frames  of  FLIR  imagery 

o  o 

have  been  digitized  from  FLER  video  tapes  at  aspects  of  0  .  -  35  , 
and  +  50°  and  spanning  ranges  from  8  kilometers  to  overflight. 

This  data  base  has  identified  a  number  of  key  issues  and  is  helping 
us  quantify  the  performance  of  the  approach  under  conditions  that 
are  typical  of  the  expected  conditions  in  the  Copperhead  and  Hellfire 
scenarios. 


REPORT  ORGANIZATION 


The  body  of  this  report  is  organized  under  the  following  headings: 

•  Section  2:  Candidate  Match  Selection 

•  Section  3:  Branch-and-Bound  Matching  Algorithm 

•  Section  4:  Criterion  Functions 

•  Section  5;  System  Simulation  and  Results 

•  Section  6:  Analysis  of  A  Priori  Information 

•  Section  1:  Minimal  Spanning  Trees 

•  Section  8:  Pattern- Matching  Data  Base 

•  Section  9;  Plans  for  the  Next  Reporting  Period 


I 


SECTION  2 


CANDIDATE  MATCH  SELECTION 


The  candidate  match  selection  algorithm  is  presented  in  this  section.  As 
shown  in  Figure  2,  the  candidate  match  selection  process  operates  on  the 
output  of  the  feature  extraction  process  and  selects  £  list  of  likely  matches 
for  the  branch- and- bound  algorithm.  It  effects  a  drastic  reduction  in  the 
complexity  of  the  problem  that  must  be  solved  by  the  branch -and- bound 
algorithm.  Part  of  this  reduction  is  achieved  by  removing  from  considera¬ 
tion  the  nondistinctive  objects  in  the  two  images.  The  distinctive  objects 
are  the  objects  that  have  high  probability  of  having  corresponding  matches 
in  the  other  image.  The  rest  of  the  reduction  is  achieved  by  rejecting 
those  object- to -object  matches  that  are  clearly  dissimilar,  as  measured 
by  the  object  features. 

The  candidate  match  selection  algorithm  in  the  current  simulation  exploits 
four  object  features.  They  are  as  follows: 

1.  Contrast  with  background 

2.  Area 

3.  Perimeter/^/ area 

4.  Minor  axis  length/major  axis  length 
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Figure  2.  Block  Diagram  of  Pattern- Matching  Process 

The  candidate  match  selection  is  done  in  two  steps: 

1.  Edit  from  the  list  of  objects  in  the  reference  image  all  but  the 

most  distinctive  objects,  using  contrast  as  a  measure  of 

distinctiveness.  Similarly,  edit  from  the  list  for  the  sensed 

image  all  but  the  N  most  distinctive  objects, 
s 

2.  For  each  remaining  object  in  the  sensed  image,  select  candidate 
matches  in  the  reference  image.  First,  remove  from  consider¬ 
ation  those  of  the  remaining  objects  in  the  reference  image  that 
have  an  area  too  large  or  too  small  or  shape  features  too 
different  to  be  matches  for  the  object  in  the  sensed  image.  Then 
use  the  area  and  shape  information  to  rank  those  objects  not 

edited.  Retain  the  C  best  matches. 

max 
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It  is  assumed  that  estimates  of  the  relative  ranges  for  the  two  images  are 


available  for  use  in  evaluating  the  quality  of  area  matches.  N  ,  N  ,  and 

s  r 

C"max  are  chosen  large  enough  to  include  the  correct  matches  with  high 
probability. 


An  example  of  Step  2  for  the  image  pair  in  Figure  3  is  shown  in  Tables  1 
and  2.  The  objects  extracted  by  the  segmentation  algorithm  are  shown  in 
Figure  4.  The  editing  process  of  Step  2  is  illustrated  in  Table  1. 
Measures  of  similarity  of  the  10  most  distinctive  objects  in  the  reference 
image  with  an  object  in  the  sensed  image  are  shown  for  the  three  features 
used  in  Step  2.  The  larger  the  measure,  the  larger  the  dissimilarity 
between  objects.  Very  dissimilar  matches  are  marked  with  asterisks, 
and  the  objects  remaining  after  editing  are  circled  file  ranking  process 
is  shown  in  Table  2.  Ranks  for  similarity  in  the  shape  features  are 
combined  into  a  shape  score.  The  shape  score  and  the  area  similarity 
rank  determine  the  overall  rank  of  an  object  for  match  quality.  The 
true  match  is  circled. 

An  example  will  illustrate  the  level  of  performance  of  the  candidate  match 
selection  algorithm.  The  candidate  match  selection  for  the  segmentations 
of  Figure  4  is  shown  in  Figure  5  and  in  Table  3.  There  are  four  correct 
object-to-object  matches  in  this  image  pair.  They  happen  to  be  the  four 
targets.  The  targets  were  among  the  five  most  distinctive  objects  in  the 
sensed  image,  and  their  correct  matches  were  ranked  either  first  or 
second  among  the  candidate  object  lists. 
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TABLE  1.  EXAMPLE  OF  EDITING  MATCHES  IN  STEP  2 

OF  THE  CANDIDATE  MATCH  SELECTION  ALGORITHM 


OBJECT  IN 
REFERENCE 
IMAGE 

CONTRAST 

RANK 

DISCREPANCY  MEASURES 

AREA 

»//A 

MINORNRAJOR 

© 

1 

.21 

M 

.28 

Sir 

2 

.71* 

.23 

.82 

141 

3 

US* 

1.78* 

.84 

4 

.13 

.72 

.18 

V 

S 

.N* 

.21 

.24 

137 

• 

.W 

1.N 

1.72* 

1S2 

7 

1.12* 

.21 

.20 

1 

.11 

.77 

M 

W 

• 

.87* 

1.88* 

1.22* 

© 

IS 

.88 

1.13 

1.13 

*  INDICATES  DISCREPANCY  MEASURE  LARGE  ENOUGH 
TO  REJECT.  ACCEPTED  MATCHES  ARE  ENCIRCLED. 


TABLE  2.  AN  EXAMPLE  OF  RANKING  ACCEPTABLE  MATCHES 


RANK  FOR  t  RANK  FOR  .  SHAPE  AREA  COMIINEO 
fjjk  BNNORANAJOR  SCORE  RANK  SCORE 


3 

2 

I 


1  4  11 

2  4  2  2 

3  4  3  3 


IN 
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CORRECT  MATCH  ENCIRCLED 


Figure  4.  Objects  Extracted  for  Images  of  Figure  3 


b.  ('.indici-i'i-  m.'iit'lics  for  i!  :<i •<>  '■  and  5 

Figure  5.  Candidate  Match  Selection  Example  (continued) 


1  1 


c.  Candidate  matches  for  (  are  <i  and 


Figure  5.  Candidate  Match  Selection  Example  (continued) 


e.  Candidate  matches  for  E  are  5  and  1.  (Object  1 
is  in  a  three-object  group  at  the  left.  ) 

Figure  5.  Candidate  Tdatch  Selection  Example  (continued) 


f.  Candidate  matches  lor  F  are  11  and  12 

Figure  5.  Candidate  Match  Selection  Example  (concluded) 


TABLE  3.  EXAMPLE  CANDIDATE  MATCH  SELECTION 


Object  in 

Sensed  Image 

Candidate  Matches 
in  Reference  Image 

A 

1,2 

B 

3,  5 

C 

6,7 

D 

6,  9 

E 

5.1 

F 

11,  12 

In  this  example,  retaining  eight  distinctive  objects  in  the  sensed  image 
and  four  candidate  matches  per  object  would  include  the  correct  matches 
with  high  probability.  For  such  a  candidate  match  selection  there  are  at 
most  17,  920  four-object  matches.  The  number  of  objects  extracted  by 
the  segmentation  algorithm  was  about  40  per  image.  This  implies  that 
before  candidate -match  selection  there  were  about  2  x  10*1  four-object 
matches.  As  will  be  seen  in  the  next  section,  the  branch-and-bound 
algorithm  finds  the  correct  match  with  about  400  match  evaluations  when 
such  a  candidate  match  selection  is  used.  On  the  other  hand,  it  is  estimated 
that  a  hardware  implementation  would  be  able  to  do  300  match  evaluations 
per  frame.  Thus,  the  present  candidate  selector  performance  appears 
very  good,  even  with  a  conservative  evaluation. 
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The  example  has  raised  a  few  key  issues  which  should  be  addressed  in 
detail.  The  first  issue  is  the  probability  of  finding  corresponding  objects 
in  the  two  fields  of  view.  In  this  example,  the  only  distinctive  objects  that 
were  common  to  both  fields  of  view  were  the  four  targets.  On  the  other 
hand,  the  background  and  foreground  clutter  has  changed  so  drastically 
that  it  is  impossible  to  identify  corresponding  clutter  objects  in  the  two 
fields  of  view.  This  is  because  the  common  ground  covered  by  the  two  fields 
of  view  can  be  extremely  small  --due  to  the  narrow  fields  of  view,  the  low 
angle  of  elevation,  and  large  aspect  angle  differences  between  the  two 
sensors.  In  Figure  6,  the  narrow  field  of  view  and  the  low  angle  of 
elevation  have  conspired  to  make  the  "footprint"  of  the  sensor  field  of 
view  on  the  ground  narrow  and  long.  For  aspect  angle  differences  as 
large  as  50°  as  in  this  example,  the  common  footprint  on  the  ground  is 
confined  to  only  the  targets. 

A  further  observation  is  that  because  of  the  low  angle  of  elevation,  clutter 
and  targets  which  possess  3D  relief  from  the  ground  (like  trees,  tanks, 
etc.  )  tend  to  be  more  invariant  to  aspect  angle  differences  than  planar 
clutter  (roads,  rivers,  temperature  gradients  in  the  terrain,  etc.  ).  This 
issue  will  be  studied  in  detail  in  the  next  reporting  period  to  determine 
the  limitations  placed  by  the  geometry  of  the  sensors  on  the  presence  of 
corresponding  distinctive  objects  in  the  two  fields  of  view.  To  maximize 
the  common  footprint,  it  appears  that  a  wide  field  of  view  look  would  be 
needed  (if  only  from  the  acquisition  sensor,  i.  e. ,  the  RPV  FLIR). 
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Figure  6.  Illustrates  the  Sensor  Geometry  for  the  Example  in  Figure  3 


(Note  the  small  common  footprint  because  of  the  low  angle  of 
elevation,  narrow  FOVs,  and  the  aspect  angle  differences.) 


The  second  issue  that  is  illustrated  by  the  example  in  this  section  is  the 
need  for  configuration  analysis.  As  we  saw,  the  candidate  match  selector 
does  not  always  rank  the  correct  object  match  highest  in  similarity,  based 
on  the  object-to-object  similarity  alone.  However,  the  correct  match  is 
within  the  first  few  most  similar  objects  for  every  object  in  the  sensed 
image.  This  points  to  the  need  for  the  configuration  matching.  In  its 
absence,  it  would  be  difficult  to  distinguish  a  specific  target  (from  all 
other  targets  and  distinctive  clutter  in  the  field  of  view)  for  target 
prioritization.  This  is  needed  to  achieve  rapid  salvo  fire,  with  successive 
projectiles  being  programmed  to  hit  different  targets  in  a  target  cluster. 
This  example  also  points  out  the  difficulty  of  truly  autonomous  acquisition, 
especially  as  it  relates  to  the  rejection  of  false  alarms. 


SECTION  3 


BRANCH-AND-BOUND  MATCHING  ALGORITHM 


The  final  search  for  the  best  multi- object  match  among  the  candidate  object- 
to- object  matches  allowed  by  the  candidate  match  selector  is  implemented 
with  a  branch- and-bound  algorithm.  It  achieves  efficiency  by  enumerating 
the  set  of  matches  in  such  a  way  that  when  a  partial  match  fails  a  simple 
test,  the  entire  subset  of  matches  containing  the  partial  match  need  not  be 
evaluated.  The  algorithm  is  presented  in  this  section  and  illustrated  with 
examples. 

The  branch- and-bound  algorithm  finds  the  best  n-object  match  that  can  be 
formed,  using  the  object- to- object  matches  allowed  by  the  candidate  match 
selector.  It  is  most  naturally  presented  as  an  efficient  way  to  search  a  tree. 

A  tree  enumerating  all  one-,  two-,  and  three-object  matches  for  the 
candidate  match  selection  of  Table  4  is  shown  in  Figure  7.  The  nodes  of 
the  tree  specify  object- to- object  matches.  An  m-object  match  is  specified 
by  the  nodes  visited  when  m- consecutive  branches  are  traversed,  starting 
from  the  root. 

In  the  branch-and-bound  search  for  the  best  match,  each  path  from  the  root 
is  traced  down  until  it  is  terminated.  A  path  is  terminated  in  two  situations: 

1.  Terminal  level:  n  branches  have  been  traced,  and  a  full  n-object 
match  has  been  specified. 
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TABLE  4 .  CANDIDATE  MATCH  SELECTION 


Object  in 

Candidate  matches 

sensed  image 

in  reference  image 

I 

1.2 

II 

3,4 

III 

5,  6 

2.  Suboptimality  Test:  The  algorithm  has  reached  a  node  which  is 
clearly  suboptimal  and  cannot  be  a  part  of  the  best  n-object  match. 

For  example,  if  the  suboptimality  test  were  satisfied  at  nodes  B  and  C 
in  Figure  7,  but  not  at  A  and  D,  a  search  for  the  best  two-object  match 
would  have  to  trace  the  branches  marked  with  bold  lines. 

The  process  of  tracing  a  path  corresponds  to  building  up  a  partial  match, 
one  object- to-object  match  at  a  time,  until  a  full  n-object  match  is  built 
or  until  it  is  clear  that  the  best  n-object  match  cannot  be  built  by  adding 
more  object- to-object  matches  to  a  partial  (<n)  object  match. 
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When  the  suboptimality  condition  occurs  at  a  node,  the  partial  match 
specified  at  the  node  is  rejected--it  cannot  be  part  of  the  best  complete 
n-object  match.  Furtherin' -re,  when  a  partial  match  is  rejected,  all 
the  complete  n-object  matches  containing  it  are  implicitly  rejected.  Rather 
than  exploring  further  implicitly  rejected  matches  subtended  by  the  node, 
the  algorithm  backtracks  to  a  node  not  previously  explored.  This  is  how 
the  algorithm  achieves  efficiency. 

For  the  suboptimality  condition  to  be  detectable,  it  is  necessary  that  the 
criterion  function  that  evaluates  the  goodness  of  a  match  satisfy  monotonicity. 
Monotonicity  is  defined  as  follows:  Let  m  be  an  object- to-object  match 
allowed  by  the  candidate  match  selector.  Let  a  match  containing  i  objeet- 
to- object  matches  be  written. 

M(1)  =  (m1,m2,...,m.). 

Let  f(M^)  be  the  function  that  evaluates  the  goodness  of  matches.  Let 
f(Mj/^)  <  f(M2^),  whenever  is  better  than  The  function 

is  defined  for  partial  matches,  as  well  as  for  complete  n-object  matches. 

Let  us  denote 

+  m  =  (mj,  m2,  ...,  m^  m). 

The  monotonicity  condition  is  satisfied  if 
f(M(i)  +  m)  2  f(M(i)). 

In  other  words,  adding  an  object- to- object  match  to  a  partial  match  never 
decreases  a  monotonic  criterion  function. 
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Using  the  monotonicity  condition  to  detect  suboptimality  is  simple.  A 
record  is  kept  of  the  best  n-object  match  found  so  far  in  the  search.  Each 
time  a  path  is  traced  to  a  new  node,  the  criterion  function  is  evaluated  for 
the  partial  match  specified  by  the  new  path.  If  the  value  obtained  is  larger 
than  the  value  of  the  criterion  function  for  the  best  n-object  match  found  so 
far,  the  monotonicity  condition  guarantees  that  all  n-object  matches  specified 
by  tracing  further  will  be  worse  than  the  current  best  match  and,  hence,  can 
be  rejected  without  explicit  evaluation.  The  best  n-object  match  formed 
during  the  search  is  obviously  the  best  n-object  match  that  can  be  formed. 


The  branch- and- bound  algorithm  is  given  in  steps  1  through  9  below.  A 
flowchart  is  shown  in  Figure  8.  The  notation  may  be  interpreted  either  in 
terms  of  a  path  being  traced  in  the  tree  or  in  terms  of  the  partial  matches 
specified  by  the  path  as  it  is  traced.  The  notation  is  as  follows: 


i  - 


ufl)~ 


-  tree  level  indicator.  The  algorithm  extends  a  path  from  the 
root  by  adding  a  branch  at  level  i.  This  corresponds  to  adding 
one  object- to-object  match  to  a  partial  match  containing  (i-1) 
object-to-object  matches. 

the  partial  match  specified  by  a  path  that  has  been  traced  i  levels 
down  the  tree.  is  an  empty  partial  match. 


(niy  }--  an  ordered  list  of  the  candidate  object-to-object  matches  at  level  i 

that  need  to  be  considered  for  addition  to  the  current  partial  match 

M^.  Equivalently,  a  list  of  the  branches  to  be  added  at  the  end 

th 

of  the  path  being  traced,  is  the  j  object-to-object  match 

in  the  list  at  level  i. 
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Figure  8.  Branch-and-Bound  Algorithm  Flow  Chart 
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nb(i)  —  the  number  of  entries  in  {m^ },  possibly  zero. 

bp(i)  --  branch  pointer  for  level  i.  Oily  one  path  is  traced  at  one 

time.  bp(i)  points  to  one  entry  in  {m. .}  that  is  in  the  path 

th 

currently  being  traced  at  the  i  level. 

M  --  the  best  n-object  match  found  so  far  in  the  search.  Initially 

>!< 

M  =  0,  the  null  match. 

>Jc 

Z  --  the  criterion  function  value  for  M  ,  initially®.  Z  is  a  bound 

>!«  >|s 

for  the  value  of  the  criterion  function  for  the  best  match  M 


The  branch-and-bound  algorithm: 


1.  Initialize 


i  =  1 
M*  =  0 


2.  Enumerate  possible  matches  for  next  level. 

Create  the  ordered  list  {m„}. 

Set  nb(i). 

3.  Preset  bp(i)  =  0. 

4.  Is  examination  of  [m^.]  complete? 

bp(i)  =  bp(i)+l. 

If  bp(i)  >  nb<i),  go  to  step  9. 

5.  Add  to  current  partial  match. 


M(i>  - 

M  =  |M  ,  m 


i,  bp(i) 
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6.  Test  for  situation  2. 

If  f(M^)  >  z.  go  to  step  4. 

7.  Extend  path  if  it  is  neither  suboptimal  nor  a  complete  n-object  match. 

If  i=n,  then  go  to  8,  otherwise 
set  i=i+l  and  go  to  2. 

8.  Update  best  match  when  a  full  match  is  formed. 

If  f(M'  ;  <  Z,  set  M  =  Mv  ' 
and  set  Z  =  f(M^).  Go  to  step  4. 

9.  When  a  subtree  has  been  completely  searched,  go  back  one  level 
to  look  for  unsearched  subtrees.  If  this  is  impossible  because 
i=l,  terminate. 

If  i>l,  set  i=i-l  and  go  to  step  4. 

If  i=l,  best  match  is  M  .  Stop. 

The  following  example  of  a  branch-and-bound  search  will  serve  to  clarify 
the  algorithm.  This  example  is  further  extended  in  the  discussion  on 
enumeration. 

To  the  right  of  and  below  each  node  in  the  tree  of  Figure  7  is  the  value  of  a 
criterion  function  for  the  match  specified  by  tracing  from  the  root  of  the 
tree  to  the  node.  Inspection  will  show  that  the  function  is  monotonic;  i.e., 
going  one  level  deeper  in  the  tree  never  decreases  its  value. 

The  branch-and-bound  search  for  the  best  two-object  match  proceeds  as 
follows:  The  first  time  step  2  of  the  algorithm  occurs,  the  tree  level 
indicator  i  is  1  and  the  six  branches  from  the  root  of  the  tree  to  nodes 
A,  B,  C,  D,  E,  and  F  are  enumerated  in  the  first  list.  Node  A  is  considered 
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first,  and  its  criterion  function  is  lower  than  the  initial  value  of  Z,  which 
is  infinity.  The  path  from  the  root  to  node  A  is  not  terminated  by  either 
of  the  two  situations  discussed. 

At  step  2  with  i=2,  the  four  branches  from  node  A  to  nodes  G,  H,  I,  and  J 
are  enumerated  in  a  second  list.  The  nodes  are  evaluated  in  turn.  When 
node  G  is  evaluated,  it  has  f(M^)  <  Z,  so  G  is  accepted  as  the  best  two- 
object  match  so  far,  and  the  value  of  Z  is  updated.  H  is  even  better  than  G; 
when  H  is  evaluated,  it  is  accepted  as  the  best  match  so  far.  I  and  J  are  not 
as  good.  After  J  is  evaluated,  the  subtree  subtended  by  A  is  completely 
searched.  This  condition  is  detected  by  the  test  of  step  4.  The  algorithm 
goes  back  to  examine  the  next  branch  in  the  first  list,  to  node  B.  At  node 
B  suboptimality  is  detected,  and  the  subtree  subtended  by  B  is  not  searched. 
The  same  thing  happens  at  node  C.  The  last  branch  to  be  examined  in  the 
first  list  is  D.  Neither  condition  terminates  the  path  to  D,  so  a  new  second- 
level  list  of  branches  from  D  to  Q  and  R  is  created.  Nodes  Q  and  R  are 
evaluated.  Q  is  a  new  best  match.  After  R  is  evaluated,  control  passes 
back  to  the  level  1  list.  Nodes  E  and  F  are  terminated  because  no  two-object 
matches  containing  the  one-object  matches  of  those  two  nodes  remain  to  be 
evaluated.  After  E  and  F  are  terminated,  the  algorithm  stops,  with  the 
match  of  node  Q  detected  as  the  best  match. 

The  two  most  important  issues  in  the  implementation  of  a  good  branch-and- 
bound  matching  algorithm  are  the  enumeration  procedure  in  step  2  and  the 
criterion  function.  The  criterion  function  will  be  discussed  in  Section  4. 


Discussion  of  the  enumeration  procedure  is  presented  below.  The  require¬ 
ments  that  the  procedure  must  meet  are  outlined,  and  the  procedure  is  presented 
in  detail.  With  the  aid  of  examples  we  can  show  how  the  procedure  meets 
the  requirements  and  what  advantages  it  has  over  alternative  procedures. 

To  guarantee  that  the  branch- and-bound  algorithm  finds  the  best  match,  it 
is  essential  that  the  enumeration  procedure  be  exhaustive,  that  is,  that  it 
be  able  to  enumerate  all  allowed  matches.  For  efficiency  the  enumerator 
should  be  nonrepetitive;  it  should  enumerate  no  allowed  match  more  than 
once.  As  will  be  seen,  it  is  also  important  that  the  enumerator  be  able  to 
prioritize,  to  order  the  lists  it  generates  to  maximize  the  probability  that  the 
correct  match  will  be  found  early  in  the  search.  The  enumerator  meets 
all  of  these  requirements.  In  addition,  it  accomodates  many- to- one  matches, 
and  it  requires  only  a  modest  amount  of  computer  memory. 

The  enumeration  procedure  has  three  steps.  In  the  first  step  an  unedited, 
unprioritized  list  is  created.  It  contains  all  matches  that  need  to  be  con¬ 
sidered  in  the  branch-and-bound  algorithm.  When  i=l,  the  list  consists  of 
all  matches  allowed  by  the  candidate  match  selector.  When  i  >  1,  the  list 
is  copied  from  the  list  for  the  previous  level.  Copying  all  of  the  previous  list 
would  guarantee  exhaustiveness,  but  not  nonrepetitiveness.  The  enumeration 
procedure  copies  all  of  the  previous  list  except  the  following: 

1.  The  object- to- object  match  in  the  current  partial  match. 

2.  Those  object- to-object  matches  which  have  already  been  considered 
at  level i-1. 

When  these  exceptions  are  made,  exhaustiveness  is  preserved  and  non¬ 
repetitiveness  is  achieved. 
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In  the  second  step  the  object- to- object  matches  in  the  new  list  that  are  not 
compatible  with  the  current  partial  match  are  removed.  This  editing 
does  not  affect  exhaustiveness.  It  is  used  to  remove  unwanted  many-to-one 
matches. 


In  the  third  step  the  list  is  reordered  so  that  the  matches  most  likely  to 
lead  to  the  correct  full  match  are  first  in  the  list.  This  prioritization 
affects  neither  exhaustiveness  nor  nonrepetitiveness.  Enumeration  is 
accomplished  by  executing  steps  1  through  3  below  at  step  2  of  the  branch- 
and-bound  algorithm.  The  notation  is  as  follows: 


i,  nb(i),  bp(i). 


m. . 


--  as  in  branch-and-bound  algorithm. 


P  (j»  i)»  P  (j»  i)  --  ordered  pair  specifying  m...  P  (j,i)  and 
s  1  ij  s 

P  (j,  i)  are  pointers  to  the  objects  in  the  sensed 

^  th 

and  reference  images  in  the  j  match  m_  in  the 

list  for  level  i. 


N 

s 

Nc<k)  - 


C(m,  k)  -- 


number  of  distinctive  objects  in  sensed  image. 

til 

number  of  candidate  matches  in  reference  image  for  k 
distinctive  object  in  sensed  image. 

th  th 

pointer  to  m  candidate  match  for  k  object  in  sensed  image. 


Enumeration  procedure: 

1.  Make  an  unedited,  unprioritized  list  of  object- to- object  matches  to 
consider  for  addition  to  the  current  partial  match.  If  i=l,  obtain 
a  list  from  the  candidate  match  selector  output.  If  i  >  1,  obtain 
a  list  from  the  list  for  the  previous  level. 
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If  i=l 


then 

begin 

j*0 

k-0 

while  k<N 

s 

k=k+l 

in=0 

while  m<N  (k) 
c 

m=m+l 

j=j+l 
Pg(j.i)=k 
pr(j.  i)=C(m,  k) 
endwhile 
endwhile 

end 

else 

begin 

j=0 

k=bp(i-l ) 
while  k<nb(i-l) 

k=k+l 

j*j+l 

Ps(j’i)=Ps(k’  i_1) 

P  (j,i)=P  (k,  i-1) 
r  r 

endwhile 


n  4 


nb(i)=j 


end 


2.  Remove  from  the  list  those  object- to- object  matches  that  are 
incompatible  with  the  current  partial  match  Mu  1 .  (Edit 
disallowed  many-to-one  matches.)  Compact  list  and  update  nb(i). 

3.  Prioritise  by  sorting  the  list  so  that  the  matches  that  appear 
most  promising  are  first  in  the  list. 

An  example  will  clarify  how  the  enumerator  achieves  exhaustiveness  and 
nonrepetitiveness.  The  tree  in  Figure  7  is  generated  by  the  enumerator 

when  no  many-to-one  matches  are  allowed  for  the  candidate  match  selection 

3  1 

of  Table  4  and  no  prioritization  is  attempted.  There  are  (  )2  =6  one-object 

3  2  3  3  1 

matches,  (2>2  =  12  two-object  matches,  and  )2  =8  three-object  matches. 

All  are  enumerated  without  repetition.  Consider  what  happens  if  suboptimality 

does  not  occur  at  node  C.  The  list  for  level  1  is  {m„}  =  {U>I2, 113,114,  III  5, 

1116} ,  and  when  the  new  list  is  enumerated  bp(l)=3.  For  bp(l)=3,  the  matches 

II,  12,  and  113  are  not  copied  to  the  new  list.  All  the  matches  containing 

II  and  12  have  been  enumerated  in  the  subtrees  of  nodes  A  and  B.  112  is 

being  considered  in  the  current  partial  match.  No  match  needed  to  form 

the  matches  containing  the  current  partial  match  that  have  not  already  been 

enumerated  has  been  left  out  of  the  new  list.  Before  editing,  the  new  list 

is  £  114,  III5,  III6}.  114  will  be  eliminated;  together  with  113,  it  would 

constitute  a  many-to-one  match.  The  matches  remaining  after  editing,  III5 

and  III6,  are  both  needed  to  form  matches  not  yet  enumerated.  By  eliminating 

from  the  new  list  all  unnecessary  matches,  but  keeping  all  the  necessary 

ones,  exhaustiveness  and  nonrepetitiveness  are  attained. 
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Good  prioritization  in  the  enumeration  procedure  causes  the  correct  match 
to  be  found  earlier  in  the  branch- and- bound  search.  This  means  that  small 
values  of  the  bound  Z  are  found  sooner  and,  thus,  more  of  the  tree  is  pruned. 
For  example,  the  search  tree  of  Figure  7  could  be  replaced  with  the  tree  of 
Figure  9  ,  if  the  first  level  list  were  prioritized.  The  number  of  two- 
object  matches  examined  in  the  new  search  tree  is  4,  two  fewer  than  in 
Figure  7.  (Note  that  all  possible  matches  are  enumerated  nonrepetitively 
in  the  tree  of  Figure  7.  The  gain  from  prioritization  is  modest  in  this 
small  example  tree.  The  gain  is  much  greater  in  a  larger  tree. 

The  present  algorithm  with  its  enumeration  procedure  compares  favorably 
with  the  alternatives  in  trading  computer  memory  and  algorithm  complexity 
for  prioritization  capability.  One  alternative  is  to  do  prioritization  of  the 
objects  to  be  matched  with  objects  in  the  sensed  image  without  prioritizing 
objects  in  the  sensed  image.  This  would  reduce  the  required  computer 
memory.  However,  an  efficient  coding  of  the  present  algorithm  requires 
only  about  400  memory  words  for  enumerating  all  five- object  matches  for 
a  candidate  match  selection  retaining  10  distinct  objects  and  5  matches 
per  object.  The  memory  that  could  be  saved  would  not  justify  the  reduced 
prioritization  capability.  The  other  alternative  is  to  adopt  an  algorithm 
that  traces  multiple  paths  simultaneously,  extending  the  most  promising 
path  first.  Such  an  algorithm  would  be  able  to  prioritize  more  and  would 
find  the  best  match  with  fewer  evaluations.  It  would  require  more  memory 
and  more  complexity,  and  for  some  problems  much  more.  The  present 
algorithm  finds  the  correct  match  in  350  to  800  evaluations,  while  it  is 
estimated  that  a  hardware  implementation  would  be  able  to  do  3000  evalu¬ 
ations  per  frame.  Experience  with  the  present  algorithm  suggests  that 
some  use  of  multi-path  techniques  might  offer  advantages  for  more  difficult 
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Figure  9.  A  Prioritized  Branch-and-Bound  Search  Tree 


problems  that  can  be  attained  with  acceptable  amounts  of  computer  memory. 
This  possibility  will  be  investigated. 


The  ordering  required  for  prioritization  is  achieved  by  evaluating  a 
heuristic  function  for  each  match  in  the  new  list  and  sorting  the  values 
obtained.  The  heuristic  function  is  the  error  in  predicting  the  position 
in  the  sensed  image,  given  the  estimate  of  the  image- to- image  transfor¬ 
mation  derived  from  the  current  partial  match.  It  will  be  discussed  in 
more  detail  in  the  next  section. 

Many-to-one  matches  do  appear  in  correct  image- to- image  matches;  it 
is  an  advantage  to  be  able  to  handle  them.  Images  sensed  at  close  range 
tend  to  have  distinctive  objects  segmented  as  multiple  objects.  At  longer 
ranges,  objects  such  as  targets  are  segmented  as  single  objects.  If  one 
image  is  at  close  range  and  the  other  at  long  range,  there  will  be  valid 
many-to-one  matches.  Changes  in  aspect  also  give  rise  to  many-to-one 
matches.  Many-to-one  matches  also  occur  with  line  structures,  as  when 
a  gap  in  a  long  line  is  bridged  by  edge-feature  extraction  algorithms  for 
one  image  but  not  the  other.  The  enumeration  procedure  offers  flexibility 
in  handling  many-to-one  matches. 
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SECTION  4 


CRITERION  FUNCTIONS 


In  this  subsection  the  criterion  function  will  be  discussed.  The  criterion 
function  must  satisfy  two  requirements: 

1.  It  must  realistically  reflect  the  goodness  of  a  match. 

2.  It  must  be  monotonic;  i.  e. ,  adding  a  match  to  a  partial  match 
must  never  decrease  the  criterion  function. 

Failure  to  meet  either  requirement  will  prevent  the  branch-and-bound 
algorithm  from  finding  the  best  match.  The  criterion  function  combines 
information  about  the  distinctiveness  of  the  objects  in  a  match  and  the 
quality  of  object-to-object  matches  with  an  evaluation  of  the  configurational 
consistency  of  the  match.  The  effectiveness  of  the  criterion  function  will 
be  discussed  in  the  subsection  on  simulation  results. 


The  criterion  function  f  has  three  components,  which  may  be  written 
as  follows: 

f(MU))  =  w  f  (M(l)) 
dist  dist 

+  w  f  (M(l)) 
cm  cm 


+  w  f  (M(0) 
conf  corn 
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where  the  w's  are  weights,  and  f ..  .,  f  ,  and  f  .  are  functions 

6  dist  cm  conf 

evaluating  the  distinctiveness  of  the  objects  matched  by  Mv  ,  the  quality 
of  object-to-object  matches  in  and  the  likelihood  that  the  configuration 

of  is  correct,  f  is  monotonic  if  the  weights  are  nonnegative  and  the 
contributing  functions  are  also  monotonic. 


The  three  functions  f ..  , ,  frrn,  and  f  ,  will  be  discussed  in  turn.  To 

dist  cx“  conf 

simplify  discussion,  let  =  (m  ,  m_, . . . ,  m. ),  and  let  s .  and  r  .  be  the 

i  *  i  J  J 

objects  in  the  sensed  and  reference  images  for  object-to-object  match  m  . 

Let  (x  .,y  .)  and  (x  .,y  .)  be  the  coordinates  of  s.  and  r.  in  the  two  images. 
S]  S3  r)Jrj  J  j 

Let  t  be  an  arbitrary  object. 


The  Distinctiveness  Component 

Given  two  matches  that  are  otherwise  as  good,  the  match  using  the  more 

distinctive  objects  is  preferred.  The  distinctiveness  component  of  the 

criterion  function,  f ..  .,  is 

dist' 

i 

tdis.<M<1,)  =I  d(8.)  +  d(r.), 

j=l 


where  d(*  )  is  a  function  that  measures  the  distinctiveness  of  the  object  it 
takes  as  an  argument,  monotonic  if  ri(*  )  is  nonnegative,  and  matches 

using  more  distinctive  objects  are  preferred  by  the  branch-and-bound 
algorithm  if  d(*  )  increases  as  objects  become  less  distinctive.  Both 
requirements  are  easily  met.  In  the  present  simulation,  d(t)  is  the 
contrast  rank  of  the  object  t. 


1 
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The  Object  Similarity  Component 


Evaluation  of  the  quality  of  object-to-object  matches  is  incorporated  into 

the  criterion  function  with  f  .  The  function  is  of  the  form 

cm 

i 


f  (m(i))  =  y 

cm  i—t 


j=l 


q(s r .). 
J  J 


f  is  monotonic  if  q(* ,  •  )  is  nonnegative.  It  favors  the  better  matches  if 

q(s.,  r.)  decreases  as  the  quality  of  the  match  between  s.  and  r.  increases. 

J  J  J  J 

In  the  present  simulation,  q(s.,  r.)  is  the  rank  of  r.  in  the  list  of  candidate 

J  J  J 

matches  for  s..  Thus,  q(r.,  s.)  is  determined  by  the  candidate  match 
3  13 

selector  operating  on  its  feature  base  without  reference  to  considerations 
of  configurational  consistency. 


The  Configuration  Component 


The  component  of  the  criterion  function  that  contributes  most  of  the  power 

to  discriminate  between  good  and  bad  matches  is  the  configuration  evaluation 

function  f  _.  f  has  three  components: 
conf  conf  r 


f  ,  (M(0)  =  f  (M(l))  +  f  (M(l))  +  f  (M(l)). 
conf  ms  topo  ap 


f  andf.  evaluate  the  self-consistency  of  the  configuration;  f 
ms  topo  ap 

evaluates  the  consistency  of  the  configuration  with  the  a  priori  information 

about  the  sensor  positions  and  orientations,  hereinafter  referred  to 

simply  as  the  a  priori  information.  Each  of  the  three  components  is 

monotonic;  therefore,  f  ,  is  monotonic. 

conf 
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fm  is  a  least -squares  residual  prediction  error  that  is  small  when  the 
positions  of  the  objects  in  the  sensed  image  may  be  predicted  accurately 
with  a  linear  transformation  operating  on  the  positions  in  the  reference 
image.  Let  W  denote  a  2  x  3  matrix.  Then 


f 

ms 


(M^)  =  min  V* 
w 


sj 

ysj 


W 


rj 

r  . 

n 


2 


An  example  showing  the  increase  in  f  as  a  match  configuration  become 

in  s  ( i ) 

less  self-consistent  is  shown  in  Figure  10.  The  optimizing  W  for  M 

* 

will  be  denoted  W..  It  is  well-known  from  linear  regression  theory  that 


* 

W. 

l 


A  B 


1 


where 


i 

j-i 


X8j 

y3j 


(x  .  y  .  1 ) 


and 


B  =  V  (x  .  y  .  1  )T  (x  .  y  .  1 ). 

4-*  rj  7r]  rj  rj 

j=l 


f  may  he  seen  to  be  monotonic  as  follows:  Let  +  *  ^  +  m. , , , 

ms  J  i-H 


Denote 


T 


REFERENCE 

CONFIGURATION 


SENSED 

CONFIGURATION 


A  perfect  match.  Points  in  the  sensed  image  are  predicted  exactly  by 
a  linear  transformation  operating  on  the  matching  points  in  the 
reference  image. 


REFERENCE 

CONFIGURATION 


.Cx 

X  • 

D  -B 


SENSED 

CONFIGURATION 


b.  A  bad  match.  The  best  predictions  for  a  linear 
transformation  are  shown  with  x  s. 

Figure  10.  An  Example  Showing  How  the  Function  f  Detects  Had  Matches 

ms 


V 


Then 


and  f  is  monotonic.  In  the  current  simulation,  prioritization  depends  on 

ms  ^ 

the  optimal  transformations  .  A  heuristic  function  g(*  )  is  evaluated  for 

the  m.  =  m..  in  {m..}  and  the  list  is  sorted.  The  heuristic  function  is: 

J  i]  iJ 


This  heuristic  function  causes  the  match  most  consistent  with  the  configur¬ 
ation  of  the  current  match  to  be  considered  next.  The  relationship  given 

>!« 

above  for  is  not  valid  for  i  <  3  because  the  matrix  B  is  singular.  For 
i  <  3  the  information  in  the  match  configuration  is  used  to  modify  an 
estimate  of  the  true  matching  transformation  based  on  the  a  priori 

knowledge.  For  example,  for  i  =  1  the  single  match  is  used  to  determine 

* 

the  translation,  and  the  other  parameters  in  are  taken  as  those  estimated 
from  the  a  priori  knowledge. 
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The  transformation  W  corresponding  to  the  best  match  may  be  used  to 
determine  the  location  in  the  sensed  image  of  any  point  in  the  reference 
image.  This  gives  the  capability  to  steer  toward  targets  that  are  not  even 
segmented  in  the  sensed  image. 


Correct  image -to -image  matches  are  topologically  consistent  regardless 
of  sensor  range,  aspect,  or  perspective.  The  function  ^  Q  evaluates  how 
much  a  configuration  must  be  altered  to  achieve  topological  consistency. 

If  the  inconsistency  is  greater  than  what  is  attributable  to  segmentation 
errors,  the  configuration  may  be  rejected. 


The  topological  consistency  function  f^  (M'  )  is  computed  as  follows: 

Let  (M  )  be  a  pair  of  triangles  in  the  sensed  and  reference  images 

defined  by  x  .,  x  ,  and  x  .  and  by  x  ,  x  .  ,  and  x  Two  examples  are 

s]  sk  si  rj  rk  rl 

shown  in  Figure  11.  The  triangle  pair  is  consistent  if  a  directed  path 
from  vertex  j  to  vertex  k  to  vertex  1  is  either  clockwise  in  both  images  or 
counterclockwise  in  both. 


Three  collinear  points  are  consistent  with  any  triangle.  The  function 

Xjki  (M  *  )  is  defined  to  be  equal  to  the  square  of  the  minimum  distance 

for  which  it  is  possible  to  move  a  point  in  either  triangle  of  A  (M^  )  and 
’  jkl 

achieve  topological  consistency.  This  is  the  square  of  the  shortest 
altitude  of  the  two  triangles.  An  example  is  shown  in  Figure  lib.  f 

topo 

combines  the  values  of  X  for  all  triangles  that  can  be  formed  with 

(i)  ^ 

M  and  is  defined  as 


n  if  v>  s  . 

f  (M  1  )  =  |  Crl' 

0^°  •  if  ^  >  1* 

cnt 
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xsl 


a.  A  topologically  consistent  match 


b.  A  topologically  inconsistent  match 
Minimum  distance  to  consistency  *  1 

Figure  11.  Examples  of  Topologically  Consistent 
and  Topologically  Inconsistent  Matches 
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where 


j=i 


max 

s  1  <  k  <  j 


Ski 


The  threshold  Mcr^t  is  chosen  so  that  an  inconsistency  greater  than  what 
is  attributable  to  segmentation  error  always  leads  to  rejection  of  M^. 
*topo  *s  monoton^c  if  i®  nonnegative,  which  it  is. 


This  method  of  evaluating  topological  consistency  has  the  advantage  that  it 
will  not  reject  configurations  with  collinear  objects  because  of  small 
segmentation  errors. 

The  function  f  (M^)  for  evaluating  the  consistency  of  a  match  with 
ap 

a  priori  information  is  defined  as 


f  (M(l))  =  f  (M(l))  +  f  (M(l)). 
ap  r  p 


f  is  computed  as  the  result  of  a  ratio  test: 


f  (M 
r 


(i) 


|0  if  r^  j.  *  r  s:  r^2!  for  1  *  k  <  j  £  i 
cnt  jk  cnt  J 

I®  otherwise. 


where 


jk 


f  is  computed  as  the  result  of  a  test  of  how  well  the  a  priori  information 
predicts  the  differences  between  points  in  the  sensed  image: 
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o  u  1 1  S\  -T  k>  -  w<xr.  -?k>  1 1  <  dcr.t 


f  =Jfor  1  s  k  <j  s  i 

Jfor  some  WeA 
V?  otherwise. 


A  is  a  set  of  matrices  consistent  with  the  a  priori  information. 


(1 )  (2)  2 

The  thresholds  r  r  ,x,  and  d  and  the  set  A  are  chosen  to  make 
cnt  crit  cnt 

the  probability  of  rejecting  an  incorrect  match  high,  while  keeping  the 
probability  of  rejecting  a  correct  match  low. 
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SECTION  5 


SYSTEM  SIMULATION  AND  RESULTS 


In  this  section  the  current  state  of  the  simulation  software  is  discussed, 
and  results  of  simulation  experiments  are  presented.  The  simulation  is 
implemented  in  three  parts.  They  are  as  follows: 

•  PATS  segmentation  algorithm 

•  Candidate  match  selector 

•  Branch- and-bound  pattern  matching  algorithm 

A  block  diagram  is  shown  in  Figure  12.  The  PATS  segmentation  algorithm 
has  been  discussed  in  a  previous  report.  *  The  algorithms  for  the  candidate 
match  selector  and  the  branch-and-bound  pattern  matcher  are  as  discussed 
in  other  sections  of  this  report.  All  the  parameters  of  the  two  algorithms 
are  inputs  to  the  actual  programs.  This  allows  full  flexibility  in  experiment¬ 
ing  with  the  parameters  of  the  algorithms. 

The  parameters  input  to  the  candidate  match  selector  are  as  follows: 

1.  N  .  N  ,  and  C  .  These  are  the  number  of  objects  to  consider 
s  r  max  J 

in  the  two  images  and  the  number  of  matches  to  retain  per  object 
as  discussed  earlier. 


P.  M.  Narendra  and  J.J.  Grabau,  Advanced  Pattern- Matching  Techniques 
for  Autonomous  Aquisition:  First  Quarterly  Progress  Report,  NV&EOL 
Contract  Number  DAAK  70-79-C-01 14,  Honeywell  Systems  and  Research 
Center,  Minneapolis,  Minnesota,  Report  Number  79SRC104,  December  197ft. 
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Figure  12.  Block  Diagram  of  Simulation  Software 

2.  Thresholds  governing  the  editing  in  step  2  of  the  algorithm. 

3.  An  estimate  of  the  ratio  of  areas  of  matching  objects  in  the  two 
images. 

The  parameters  N  ,  N  ,  and  C  and  the  thresholds  may  be  set  large  to 
s  i*  max 

simulate  degraded  candidate  match  selector  performance. 

The  candidate  match  selector  program  has  a  display  capability  for  interactive 
experimentation  with  the  algorithm. 
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The  parameters  input  to  the  branch-and-bound  matching  programs  are 
as  follows: 

1.  The  weights  w,.  ,,  w  ,  and  w  .  for  the  three  main  components 

6  dist  cm  conf  r 

of  the  criterion  function. 

2.  Thresholds  for  the  functions  f.  ,  f  ,  and  f  . 

topo  r  p 

3.  An  estimate  of  the  matching  transformation. 

4.  A  specification  of  whether  or  not  to  prioritize. 

By  setting  weights  and  thresholds  appropriately,  it  is  possible  to  inhibit 
the  different  features  of  the  criterion  function  and  determine  their  con¬ 
tributions  to  the  power  to  reject  incorrect  matches.  The  effect  of  errors 
in  estimating  the  matching  transformation  from  the  a  priori  information  may 
also  be  determined. 

Outputs  of  the  program  that  are  used  in  algorithm  evaluation  are  as  follows: 

1.  An  image  display  that  shows  the  matches  evaluated  and  the  best 
match  in  a  simple  color- coded  format. 

2.  A  CRT  output  of  the  values  of  the  three  components  of  the  criterion 
function  and  the  thresholding  functions  for  which  a  threshold 

was  crossed,  for  use  with  (1)  above. 

3.  A  hard-copy  printout  that  includes  the  information  of  (2),  plus  a 
trace  of  the  tree  enumerated  and  statistics  on  the  number  of  match 
evaluations  at  each  level  of  the  tree. 


These  outputs  give  a  detailed  picture  of  algorithm  performance 


Three  examples  of  simulation  results  will  be  presented  in  this  section. 

The  main  purpose  of  the  first  example  is  to  show  the  power  of  the  a  priori 
information  in  rejecting  incorrect  matches.  The  second  example  demon¬ 
strates  the  ability  of  the  algorithm  to  find  the  correct  match  quickly  after 
it  has  found  a  correct  object- to- object  match  at  level  one  of  the  tree.  This 
capability  exists  because  of  the  full  prioritization  capability  of  the  enumeration 
procedure.  The  third  example  shows  how  the  power  of  a  test  for  the  con¬ 
sistency  of  configurations  varies  with  the  accuracy  of  the  a  priori  information. 


EXAMPLE  1 

The  main  point  of  this  example  is  the  power  of  the  a  priori  information  for 
rejecting  bad  matches.  The  candidate  match  selection  for  this  example  is 
the  one  shown  in  Section  2  in  Figure  6  and  Table  3.  Algorithm  parameters 
are  as  shown  in  Table  5.  The  full  search  tree  is  shown  in  Figure  13. 

The  correct  match  was  determined  with  49  match  evaluations.  The  nodes 
of  the  tree  were  examined  in  the  order  indicated  by  the  numbers  to  the  right 
of  and  below  the  nodes.  The  first  14  match  evaluations  are  shown  in  Figure 
14.  The  objects  involved  in  a  match  are  marked  with  the  symbols  of  Table  4. 
The  node  numbers  of  Figure  13  are  at  the  upper  left  in  the  photographs  of 
Figure  14.  The  photographs  at  the  top  are  for  the  sensed  image;  those  at 
the  bottom  are  for  the  reference  image. 

The  algorithm  starts  with  an  incorrect  match  at  node  1.  It  enumerates  two- 
object  matches  containing  this  match  at  nodes  2  through  7.  All  of  those 
matches  can  be  rejected  easily,  using  only  the  tests  of  consistency  with  the 
estimate  of  the  matching  transformation  derived  from  the  a  priori  information. 
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TABLE  5.  BRANCH-AND-BOUND  ALGORITHM  PARAMETERS 
FOR  EXAMPLE  1 


Weights 


Estimate  of 
Matching  Trans¬ 
formation  from 
a  priori 
information 


Prioritization 


W  =  W  =  W 
dist  cm  conf 


u  ..  =10  pixels 
crit 


r  -  70%  of  t rue  sea  it¬ 
er  it 


r  ..  =  130%  of  true  scale 

crit 


d  r  50  pixt-ls 
crit 


10  roll  error,  translation 
error  equal  to  distance  between 
targets 


Inhibited 


When  all  the  two-object  matches  are  rejected,  the  search  of  the  subtree 
subtended  by  node  1  is  complete,  and  the  algorithm  passes  to  the  next 
match  at  level  one  of  the  tree  which  is  at  node  8.  This  is  a  correct 
match.  Two-object  matches  containing  this  match  are  enumerated  at 
nodes  9  and  22  through  26.  The  first  considered  is  correct.  Three-object 
matches  containing  the  correct  two- object  match  specified  at  node  9  are 
enumerated  at  nodes  10  through  13.  The  incorrect  mutches  of  nodes  10 
through  12  are  easily  rejected,  using  only  the  tests  of  consistency  with  a 
priori  information.  The  correct  three-object  match  is  found  at  node  13. 
The  four-object  matches  containing  this  match  are  enumerated  at  nodes  14 
through  17.  The  match  at  node  14  is  the  correct  four-object  match. 
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Figure  14.  The  First  14  Mutc  hes  Evaluated  in  Example  1 
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Figure  14.  The  First  14  Matches  Evaluated  in  Example  1  (continued) 


Figure  14.  The  First  14  Matches  Evaluated  in  Example  1  (continued) 
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Figure  14.  The  First  14  Matches  Evaluated  in  Example  1  (continued) 
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Figure  14.  The  First  14  Matches  Evaluated  in  Example  1  (continued) 
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Figure  14.  The  First  14  Matches  Evaluated  in  Example  1  (continued) 
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Figure  14.  The  First  14  Mutches  Fvn  hinted  in  F.snmple  1  (continued) 

i, 


n.  \od<*  14. 


,n  i  'A  i :  r  1 


Figure  14.  The  First  14  Matcht-s  Kvaluat 


pie  1  (concluded) 


In  the  other  simulation  experiments  it  was  found  that,  as  in  the  example, 
most  of  the  incorrect  matches  could  be  rejected  at  high  levels  in  the  tree 
by  checking  for  consistency  with  the  a  priori  knowledge.  Munition  roll 
will  probably  be  the  most  important  perturbation  in  the  a  priori  information. 
Simulated  errors  in  roll  measurement  of  10°  had  no  effect  on  the  matching 
process  for  this  example. 

EXAMPLE  2 

This  example  shows  that  the  prioritization  capability  makes  it  possible  for 
the  algorithm  to  find  the  correct  match  quickly.  The  candidate  match  selection 
for  this  example  is  shown  in  Table  6.  The  parameters  of  the  candidate 
match  selection  algorithm  were  set  to  simulate  a  lower  selection  performance 
than  that  of  example  1.  The  branch-and-bound  parameters  were  as  in 
example  1,  except  prioritization  was  enabled.  The  first  four  matches 
enumerated  are  shown  in  Figure  15  in  the  same  format  as  for  example  1. 

The  correct  four-object  match  was  found  in  the  minimum  time- -four  match 
evaluations. 

EXAMPLE  3 

In  this  example  data  is  presented  to  show  how  the  power  of  one  of  the  two 

tests  of  consistency  with  the  a  priori  knowledge- -the  prediction  quality  test 

f  --depends  on  the  threshold  d  » . .  The  candidate  match  selection  was  as 
p  r  crit 

in  example  2.  The  branch-and-bound  algorithm  parameters  were  as  in 

example  1.  except  d  was  varied.  The  number  of  match  evaluations 
v  crit 

for  the  different  levels  of  the  tree  are  shown  for  three  different  values  of 

d  in  Table  7.  The  test  f  operates  at  all  levels  of  the  tree  except  the 
crit  P 

first.  It  has  no  effect  on  the  number  of  matches  evaluated  at  levels  1  and  2 
of  the  tree.  Table  7  shows  that  there  is  a  large  reduction  in  the  number  of 
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TABLE  6.  CANDIDATE  MATCH  SELECTION  FOR  EXAMPLE  2 


Object  in 
sensed  image 

Candidate  matches 
in  reference  image 

A 

1,  2,  3,  4 

B 

5,  3,  2,  1 

C 

6,  7,  8,  9 

D 

6,  9 

E 

5,  1,  10,  3,  2 

F 

11.  12,  3,  13,  2 

C. 

1,  2,  3,  4 

H 

3,  11,  2,  13 

matches  evaluated  at  levels  3  and  4  of  the  tree  when  the  test  is  switched 

from  d  .  =  ®  (disabled)  to  d  .  =  100  pixels, 
crit  crit  r 

The  reduction  occurred  even  though  many  of  the  incorrect  matches  were 
rejected  by  the  other  tests,  and  even  with  a  value  of  d  as  large  as  100 
pixels.  The  conclusion  is  that  f^  is  powerful,  even  with  a  high  d^.^.  This 
means  that  extreme  high  quality  a  priori  information  is  not  essential  for 
f  to  be  powerful. 
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Figure  15.  The  First  Four  M.i1<  h<  ,  I  ;vr  1  u-<1  <  ■  !  m  Hxnm  pie 
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b.  Second  in  nr!,.  j .  .■  i  ’ !  i  ’  .:u!  I'  with 

Figure  15,  The  First  1’our  Mutches  Evaluated  in  Example  2 
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(continued) 
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d.  Fourth  match.  A  is  nviU-h.-t:  ••  ith  >,  I  '  v.  i!|t  i  ;  with  5,  and 

This  i  s  !!,'  •  r  a  ;  'in 


with  9 


Figure  15.  The  First  Four  Matches  Evnluntr  <1  in  Example  2  (concluded) 


TABLE  7.  DATA  FOR  EXAMPLE  3 


Level 

d  .  =  ® 
crit 

d  =  100  pixels 

crit 

d  .  =  50  pixels 
crit 

1 

19 

19 

19 

2 

219 

219 

219 

3 

375 

131 

77 

4 

158 

37 

37 

771 

406 

342 

75 


SKUTiON  6 


ANALYSIS  OF  A  PRIORI  INFORMATION 


It  has  been  found  experimentally  that  the  a  priori  information  is  powerful 
in  rejecting  incorrect  matches.  In  this  section  a  simple  theoretical 
analysis  of  the  power  of  the  ratio  test  for  two- object  matches  is  presented. 
Then  other  work  initiated  toward  full  exploitation  of  the  a  priori  information 
is  described. 

The  ratio  test  has  the  advantage  that  it  may  be  used  at  a  high  level  in  the 
tree — the  second  level.  The  least-squares  consistency  test,  by  a  comparison, 
has  no  discriminating  power  until  the  search  has  reached  the  fourth  level 
of  the  tree. 


The  analysis  of  the  power  of  the  ratio  test  is  as  follows:  Assume  that  the 
coordinates  of  the  objects  are  (xgl,  y  j)  and  (xg2„  yg2>  in  the  sensed  image 
and  (x  ,  y  ,)  and  (x  ,  y  )  in  the  reference  image.  Assume  that  all  the 
x's  and  y's  are  statistically  independent  zero-mean  Gaussian  random  variables 
normalized  to  have  variance  §.  This  assumption  is  the  key  to  the  simplicity 


of  the  analysis.  The  assumption  of  statistical  independence  between  images 
corresponds  to  the  situation  that  the  match  is  incorrect.  Define 


Z 

r 


'W'  +<ysrys2 


<XrrXr2)  +0'rrJ'r2 


) 

) 


2 
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Zg  and  Z  are  independent  chi-square  variables  with  two  degrees  of  freedom. 
The  ratio  test  is  of  form 


Accept  match  if  r..  Z  ^  Z  ^  r„Z  . 

x  r  s  u  r 


The  probability  of  accepting  is 

r  r~S 

Pr  (accept) 


■/  y 

o  r^s 


-s  -t  ,,  , 
e  e  dt  ds 


(r2_rl) 


(rj+1)  (r2+l) 


The  smaller  the  acceptance  range,  the  lower  the  probability  of  accepting  an 
incorrect  match.  The  narrowness  of  the  range  that  may  be  specified  depends 
on  the  quality  of  the  a  priori  information. 


A  conservative  analysis  shows  that  for  RPV  looks  and  munition  approaches 
with  a  depression  angle  of  45°,  one  would  be  able  to  do  at  least  as  well  as 


<FOV  \2 

mr) 


where  FOV  and  FOV  are  the  fields  of  view  for  the  sensed  and  reference 
s  r 

image.  For 

FOV  =  3° 
s 

FOV  =15° 
r 

this  gives 

P^accept)  =  5.4% 


j 

*1 
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The  ratio  test  will  be  even  more  powerful  for  three- object  matches.  As  a 
comparison,  tests  of  topological  consistency  cannot  reject  two-object 
matches,  and  symmetry  arguments  show  that  they  reject  only  approximately 
one  half  of  all  incorrect  three-object  matches. 

It  is  clear  that  there  are  more  powerful  ways  to  exploit  the  a  priori 
information  than  the  simple  ratio  test.  For  example,  a  modification  that 
detected  180°  reversals  would  do  twice  as  well.  To  find  the  best  way  to 
exploit  the  a  priori  information,  a  powerful  program  is  being  written  to 
analyze  how  errors  in  estimation  of  the  sensor  orientation  and  positions 
affect  the  matching  transformation.  This  program  is  near  completion. 

Its  important  features  include  the  following: 

•  It  allows  parameters  to  be  specified  and  analyzed  in  their  most 
natural  coordinate  system.  For  example,  the  position  of  an  RPV 
with  respect  to  the  GCS  and  the  position  of  a  projectile  with  respect 
to  its  cannon  may  be  most  meaningfully  analyzed  in  two  different 
coordinate  systems. 

•  It  supports  analysis  for  different  matching  techniques;  for  example, 
ground-plane- to- ground- plane  or  sensor-plane-to-sensor-plane. 

•  Any  linear  combination  of  16  true  parameters  or  16  estimated 
parameters  relevant  to  the  problem  may  be  specified  as  a 
perturbation  to  be  analyzed. 

•  The  matching  transformation  is  calculated  in  a  standard  matrix  form 
and  also  in  a  unique  skew  form  that  isolates  the  effect  of  errors  in 
the  munition  roll,  one  of  the  most  variable  parameters. 
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The  skew  form  of  a  transformation  matrix  W  is  specified  by  the  six  parameters 

61*  ei*  e2*  62*  W13*  an<J  W23  0131  satisfy  1156  ecluations  below.  In  sensor- 
plane- to- sensor-plane  matching  as  in  the  current  simulation,  the  roll  of  the 
munition  is  merely  a  variation  in  the  single  parameter  6^. 


The  transformation  W  =  [Ab]. 


where  A  = 

cos 8^  -  sin@^ 

—  — 

el  ° 

cos82  -  sin82 

sin8j  -  cos81 

°  e2_ 

sin02  -  cos02 

and 


b  = 


W 


13 


W, 


23 


A  simple  noniterative  algorithm  for  calculating  the  skew  form  of  a  matrix 
has  been  implemented  for  use  in  the  analysis  program. 


SECTION  7 


MINIMAL  SPANNING  TREES 


As  we  saw  in  the  previous  sections,  the  branch-and-bound  algorithm 
considers  sequentially  object  matches  to  find  an  optimal  match  subset. 

One  way  to  speed  up  the  convergence  to  the  optimal  configuration  match 
would  be  to  match  distinctive  clusters  of  objects  rather  than  individual 
objects.  The  branch-and-bound  algorithm  can  then  match  precisely 
the  configurations  of  individual  objects  within  the  clusters.  In  fact, 
this  idea  corresponds  to  gestalt  cognition  phenomena  encountered  in 
human  perception.  A  further  advantage  of  matching  clusters  is  that 
extended  objects  like  roads,  rivers,  etc.,  are  often  apt  to  be  segmented 
into  multiple  components.  Matching  clusters  of  these  components  between 
the  two  fields  of  view  is,  therefore,  likely  to  be  more  robust. 

The  basic  idea  is  to  initially  match  distinctive  clusters  in  the  two  fields 
of  view.  Several  clustering  algorithms  exist  for  partitioning  data  points 
in  a  multi-dimensional  space.  However,  a  data  structure  known  as 
minimal  spanning  trees  offers  the  highest  promise  for  extracting  distinctive 
clusters  in  this  application.  The  following  is  a  brief  discussion  of  minimal 
spanning  trees  and  the  results  of  initial  experiments  we  have  performed 
on  the  test  images  described  in  the  previous  sections. 

The  coordinates  of  objects  extracted  from  an  image  form  a  set  of  points. 

A  complete  graph  for  such  a  set  of  points  has  an  edge  linking  each  pair 
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of  points  as  shown  in  Figure  16a.  A  minimal  spanning  tree  consists  of 
the  subset  of  edges  in  the  complete  graph  that  minimizes  the  total  length 
of  its  edges  and  still  provides  a  path  between  any  pair  of  points.  The 
minimal  spanning  tree  for  the  graph  of  Figure  16a  is  shown  in  Figure  16b. 

Traditionally,  minimal  spanning  trees  have  been  used  in  cluster  analysis 
as  follows:  A  minimal  spanning  tree  is  constructed  for  the  points  specified 
by  the  data  to  be  clustered.  Then  the  longest  edges  in  the  minimal 
spanning  tree  are  deleted.  If  the  data  really  clusters,  this  process  will 
break  the  minimal  spanning  tree  into  groups  of  connected  points  corres¬ 
ponding  to  the  clusters,  as  shown  in  Figure  17. 


a.  Complete  graph 


b.  Minimal  spanning  tree 
Figure  16.  A  Minimal  Spanning  Tree  for  a  Graph 


Figure  17.  A  Minimal  Spanning  Mat  Clusters  When  it  is  Broken 

This  procedure  could  be  used  to  group  the  distinctive  objects  in  image  pairs 
into  clusters.  The  minimal  spanning  trees  would  be  constructed  for  the 
distinctive  objects  in  the  two  images  and  then  broken  to  obtain  clusters. 
Software  was  written  to  test  this  concept  on  real  images.  Examples  of 
results  for  the  images  of  Figure  3  are  shown  in  Figure  18.  The  distinctive 
objects  are  indeed  clustered  into  groups.  Furthermore,  for  this  image 
pair  it  would  be  easy  to  determine  that  the  clusters  containing  the  very 
distinctive  targets  should  be  matched. 
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Minimal  spannii,;  •  •  ,  .  *■  :  ’7  i 


Figure  18.  Minimal  Spanning  »  in  Iinupc.s  of  Figure 


Figure  18.  Minimal  Spanning  Trees  in  Images  of  Figure  3  (concluded) 


SECTION  8 


PATTERN-MATCHING  DATA  BASE 


As  we  have  seen  in  the  previous  sections,  the  pattern-matching  algorithms 
must  successfully  match  images  from  a  variety  of  sensor  geometries  and 
sensor  types.  For  the  simulation  task,  ideally,  we  need  images  of  the 
same  scene  from  FLIR  and  TV  sensors,  representing  perspective  changes 
from  0  to  180  deg,  aspect  changes  of  1  to  90  deg,  and  a  variety  of  illumina¬ 
tion  conditions.  This  section  reports  the  status  of  the  continuing  pattern¬ 
matching  data  base  acquisition  task. 

In  this  reporting  period,  we  have  expanded  significantly  the  pattern- matching 
data  base  by  digitizing  36  frames  from  the  PATS  training  video  tapes 
generated  at  AP  Hiil  by  NV&EOL.  The  frames  are  from  three  runs 
(at  altitudes  of  500  ft  and  750  ft)  at  approach  angles  of  -35°,  0°,  and  +50° 
over  four  stationary  targets  in  a  formation.  The  digitized  frames  represent 
ranges  from  8  km  to  overflight.  Samples  from  these  frames  are  shown  in 
Figure  19.  Note  that  because  of  overcast  conditions,  ground  clutter  is  not 
conspicious,  and  the  targets  are  in  fact  the  most  distinctive  objects  in  the  FOV. 

A  promising  new  source  of  pattern-matching  imagery  is  from  the  March  1980 
PATS  flight  tests  at  AP  Hill.  These  tapes  contain  both  wide  and  narrow 
fields  of  new  imagery  and  also  represent  a  wider  range  of  background 
clutter  conditions  because  of  changing  cloud  cover  and  other  temporal 
variations.  These  tapes  are  now  being  reviewed  to  select  references  which 
may  provide  meaningful  characterization  of  the  pattern-matching  algorithm. 
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Perspective  angle  -35‘ 
Range  1 


Perspective  angle  -35° 
Range  2 


Figure  19.  Some  FLIR  Images  from  the  AP  Hill  Data  Base 
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Perspective  angle  -35 
Range  7 


Perspective  angle  -35° 

Range  8 

Figure  19.  Some  FLIR  Images  from  the  AP  Hill  Data  Base  (continued) 


Perspective  angle  0° 
Range  12 


Figure  19. 


Some  FLIR  Images  from  the  AP  Hill  Data  Base  (continued) 


Perspective  angle  50° 
Range  1 


Perspective  angle  50° 

Range  2 

Some  FLIR  Images  from  the  AP  Hill  Data  Base  (continued) 
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Perspective  angle  50° 
Range  5 


M 


Perspective  angle  50° 

Range  6 

Figure  19.  Some  FLIR  Images  from  the  AP  Hill  Data  Base  (concluded) 
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To  acquire  TV  and  FLIR  imagery  of  the  same  scene,  we  are  reviewing  the 
Advanced  Target  Tracker  video  tapes  (also  from  NV&EOL)  to  select 
appropriate  sequences.  Unfortunately  die  task  is  being  somewhat  hampered 
by  the  malfunctioning  of  the  875-line  Westel  recorder. 

Table  8  summarizes  the  current  state  of  the  pattern- matching  data  base. 


TABLE  8.  CURRENT  DATA  BASE  SUMMARY 


SOURCE 

TYPE 

FRAMES 

PERSPECTIVES 

RANGE 

PATS 

Training 

FLIR 

875  line 

36 

0°,  -35°,  +50° 

0-8  km 

LOHTADS 

(SWISTAK) 

FLIR 

525  line 

6 

*15° 

3-4  km 
to  closure 

TERRAIN 

MODEL 

Visual 

16 

0°,  90° 

- 
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SECTION  9 


PLANS  FOR  THE  NEXT  REPORTING  PERIOD 


From  the  results  reported  in  the  previous  sections,  it  is  apparent  that 
although  we  have  demonstrated  successful  pattern  matching  on  FLIR 
examples,  we  still  need  to  quantify  the  algorithm  performance  under  a 
systematic,  controlled  set  of  sensor  and  scene  parameters.  Quantification 
of  the  performance  can  be  divided  roughly  into  two  stages.  The  first 
fundamental  issue  is  the  probability  of  finding  unique,  distinctive  objects 
in  the  two  fields  of  view  which  can  be  used  in  the  matching.  This  is  a 
function  of  the  field  of  view,  the  differences  in  the  perspective  angles, 
the  angle  of  elevation,  and  the  scene  itself. 

Second,  given  the  presence  of  corresponding  objects,  how  quickly  does  the 
branch-and-bound  algorithm  find  the  optimal  corresponding  matches  ? 

This  is  a  function  of  not  only  the  search  procedure,  but  also  the  fidelity  of 
the  criteria  used  to  evaluate  a  good  match.  The  performance  of  the  branch- 
and-bound  algorithm  obviously  determines  the  computational  requirements 
for  its  implementation  in  the  munition. 

Accordingly,  our  plans  for  the  next  reporting  period  are  as  follows: 

•  Quantify  the  probability  of  finding  corresponding  objects  in  the  two 
fields  of  view  as  a  function  of  the  target  signature,  background 
signatures,  sensor  fields  of  view,  perspective  and  angle  of  elevation 
differences,  FLIR,  TV  sensors,  etc. 
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•  Given  the  corresponding  objects  to  be  matched,  quantify  experimentally 
the  performance  of  the  algorithm  and  the  criterion  function  in 

terms  of  the  number  of  subsets  evaluated  and  the  optimality 
of  the  final  result  for  various  aspect  angles.  Determine  the  sensi¬ 
tivity  of  the  algorithm  to  segmentation  noise  and  the  absence  of 
corresponding  objects  in  the  two  fields  of  view. 

•  To  provide  the  data  for  the  above  validation,  digitize  the  imagery 
representative  of  the  semi- autonomous  acquisition  scenario 
(especially  wide  and  narrow  fields  of  view)  from  the  875-line  video 
tapes  newly  acquired  from  the  PATS  flight  tests. 

•  Digitize  frames  from  video  tapes  of  the  same  scenes  from  the  TV 
and  FLIR  sensors,  gathered  by  NV&EOL  for  the  Advanced  Target 
Tracker  Program.  This  will  test  the  algorithms  under  different 
spectral  characteristics. 

•  Explore  additional  techniques  to  make  the  branch -and -bound 
algorithm  computationally  more  efficient.  As  we  saw  in  Section  5, 
the  optimal  subset  is  obtained  early  in  the  search  process,  although 
the  majority  of  the  search  time  is  spent  in  validating  the  optimality 
of  this  match.  Therefore,  it  is  feasible  to  trade  off  the  guarantee 
of  the  optimality  for  the  increased  computational  efficiency  of  the 
algorithm. 
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